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ABSTRACT 


This  paper  examines  a  family  of  program  test  data  selection 
criteria  derived  from  data  flow  analysis  techniques  similar  to 
those  used  in  compiler  optimization.   It  is  argued  that  currently 
used  path  selection  criteria  which  examine  only  the  control  flow 
of  a  program  are  inadequate.   Our  procedure  associates  with  each 
point  in  a  program  at  which  a  variable  is  defined,  those  points 
at  which  the  value  is  used.   Several  related  path  criteria,  which 
differ  in  the  number  of  these  associations  needed  to  adequately 
test  the  program,  are  defined  and  compared. 


•  yT~o 


Introduction 

Program  testing  is  the  most  commonly  used  method  for  demonstrating  that  a 
program  actually  accomplishes  its  intended  purpose.  The  testing  procedure  consists 
of  selecting  elements  from  the  program's  input  domain,  executing  the  program  on 
these  test  cases,  and  comparing  the  actual  output  with  the  expected  output  (in  this 
discussion,  we  assume  the  existence  of  an  "oracle",  that  is,  some  method  to 
correctly  determine  the  expected  output).  While  exhaustive  testing  of  all  possible 
input  values  would  provide  the  most  complete  picture  of  a  program's  performance, 
the  size  of  the  input  domain  is  usually  too  large  for  this  to  be  feasible. 
Instead,  the  usual  procedure  is  to  select  a  relatively  small  subset  of  the  input 
domain  which  is,  in  some  sense,  representative  of  the  entire  input  domain.  An 
evaluation  of  the  performance  of  the  program  on  this  test  data  is  then  used  to 
predict  its  performance  in  general.   Ideally,  the  test  data  should  be  chosen  so 
that  executing  the  program  on  this  set  will  uncover  all  errors,  thus  guaranteeing 
that  any  program  which  produces  correct  results  for  the  test  data  will  produce 
correct  results  for  any  data  in  the  input  domain.  However,  discovering  such  a 
perfect  set  of  test  data  is  a  difficult,  if  not  impossible  task  [1,2].   In 
practice,  test  data  is  selected  to  give  the  tester  a  feeling  of  confidence  that 
most  errors  will  be  discovered,  without  actually  guaranteeing  that  the  tested  and 
debugged  program  is  correct.  This  feeling  of  confidence  is  generally  based  upon 
the  tester's  having  chosen  the  test  data  according  to  some  criterion;  the  degree  of 
confidence  depends  on  the  tester's  perception  of  how  directly  the  criterion 
approximates  correctness.  Thus,  if  a  tester  has  a  "good"  test  data  criterion,  the 
problem  of  test  data  selection  is  reduced  to  finding  data  that  meet  the  criterion. 

One  class  of  test  data  selection  criteria  is  based  on  measures  of  code 
coverage.   Examples  of  such  criteria  are  statement  coverage  (every  statement  of  a 
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program   must    be   executed    at    least    once    during   testing)    and    branch    coverage    (every 
branch   must    be    traversed).      Other  coverage,  measures    Include   Cn    coverage 
measures    [3],    TERn  measures    [U1  and   boundary-interior    t5].      Obviously,    once   such   a 
criterion   has   been   chosen,   test   data  must    be   selected    to   fulfill   the   criterion. 
One  way   to   accomplish   this   is   to  select   paths    through   the  progrcim  whose  elements 
fulfill   the  chosen   criterion,   and    then   to    find    the   input  data  which  would   cause 
each  of  the  chosen  paths   to  be   selected . 

Using  path   selection  criteria  as  test  data   selection  criteria  h«is  a  distinct 
weakness.      Consider  the   strongest   path   selection  criterion  which   requires   that  all 
program  paths   p.,p^,...    be   selected.      This  effectively  partitions   the   Input  domain 
D   Into   a   set   of  classes  D=\^D[J3   such    that   for   every  x€d,    x£d[J]   if  and   only  if 
executing  the  program  with   input  x   causes  path   p  .   to   be   traversed .      Then   a   test 
T={t,  ,t„  ,. . .  1,   where   t     £D[j],  would   seem   to   be  a  reasonably   rigorous    test    of  the 
program.      However,   this  still  does  not  guarantee  program  correctness.      If  one  of 
the  D[J]   is  not  revealing   [2],   that   is    for  some  x    €D[J]   the  program  works 
correctly,    but    for  some  other  XjCDLj]   the  program   is  incorrect,   then    if  x     is 
selected   as   t.   the  error  will  not  be  discovered.      In  figure    1  we  see  an  example  of 
this.      Two  test  cases   would  be   sufficient   to   execute  all   paths   in  this   program.      If 
the  two   test  values  chosen   for  x  are   2  and   5,   then  we  would  not  discover  that  the 
condition   "if  x   >    3"   should,    in   fact,   have  been  "if  x  ^  3".      This   problem   is 
compounded   further  by  the  fact  that  many  programs  have  a  very  large,  or  possibly 
infinite,  number  of  paths  and   thus   the  criterion  that  all  paths  be   selected   must   be 
replaced   by  a  significantly  weaker  criterion  that  selects  only  a  subset  of  the 
paths . 

Although   we   must    be  aware   that   path   selection   criteria   cannot  insure   that  a 
set   of  test   data   capable  of  uncovering  all   errors  will   be   chosen,   we  are   not 
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arguing  that  such  criteria  be  abandoned.   In  the  absence  of  feasible  and  reliable 
methods  to  formally  prove  correctness  for  all  programs,  we  must  continue  to  use 
testing  strategies.   Developing  adequate  path  selection  criteria  will  help  bring  us 
closer  to  establishing  correctness.   In  [6]  the  reliability  of  path  analysis  is 
demonstrated.   Furthermore,  path  selection  criteria  are  used  to  determine 
correctness  by  symbolic  execution  of  the  code  [7,8,9,10].   Our  main  goal  for  path 
selection  criteria  is  that  the  number  of  paths  selected  be  small  enough  so  that  all 
tests  can  be  completed,  yet  large  enough  so  that  we  can  uncover  many  errors.  In 
addition,  we  want  criteria  that  can  be  checked  for  mechanically.  That  is,  we 
should  be  able  to  write  a  program  that,  given  as  input  a  program,  a  set  of  test 
data,  and  a  path  selection  criterion,  will  tell  us  whether  the  program  paths  that 
would  be  executed  using  the  test  data  are  sufficient  to  satisfy  the  criterion.  In 
addition,  this  program  should  also  be  able  to  give  us  some  indication  as  to  why  a 
given  set  of  test  data  is  inadequate.  Of  course,  we  would  also  like  to  be  able  to 
use  the  path  selection  criteria  to  mechanically  generate  a  set  of  paths  that  meet 
the  criterion  and/or  a  set  of  test  data  for  a  given  program,  but  that  is  a 
difficult,  and  sometimes  Impossible,  task. 

Most  path  selection  criteria  are  based  on  control  flow  analysis,  which 
examines  the  branch  and  loop  structure  of  a  program.  We  believe  that  data  flow 
analysis,  which  is  widely  used  in  code  optimization  [11],  should  be  considered  as 
well.  Data  flow  analysis  focuses  on  how  variables  are  bound  to  values,  and  how 
these  variables  are  to  be  used.  Rather  than  selecting  progreun  paths  based  solely  on 
the  control  structure  of  a  program,  the  data  flow  criteria  presented  in  this  paper 
track  input  variables  through  a  program,  following  them  as  they  are  modified,  until 
they  are  ultimately  used  to  produce  output  values.  These  criteria  are  constructed 
so  that  critical  associations  between  the  definition  of  a  variable  and  its  uses  are 
examined  during  program  testing.   It  Is  our  belief  that,  just  as  one  would  not  feel 
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confident  about  a  program  without  execiitiTig  every  statement  in  it  as  part  of  some 
test,  one  should  not  feel  confident  aboat  a.  program  without  having  seen  the  effect 
of  using  the  value  produced  by  each  and  every  computation. 

In  the  next  section  we  present  a  programming  language  and  define  some  graph- 
theoretic  terminology.  We  then  introduce  a  hierarchy  of  path  selection  criteria 
based  on  control  and  data  flow  analysis  of  a  program.   In  the  last  section  we 
discuss  the  relative  strengths  and  weaknesses  of  the  criteria. 

The  Programming  Language 

He  now  introduce  our  formal  programming  language.  This  may  be  thought  of  as 
either  the  intermediate  level  language  produced  by  compilation  from  a  high  level 
language  or  the  actual  language  in  which  the  prograun  was  written.  Our  language 
allows  only  simple  variables  and  contains  the  following  legal  statement  types; 
Start  statement:  start 


Input  statement:  read  x  , . . .  ,x 

where  x,  ,...,xr  are  variables. 

1  n 


Assignment  statement:     y:  =  f  (xj^, . . .  ,x^) 

where  f  is  an  n-ary  function   (n  >^  0)   and  y,x  ,...,x     are  variables. 

Output  statement:      print  e-^^,.,.fe^ 

where  e^^  ,   i=1,...,n,   is  either  a  literal  or  a  variable. 

Unconditional  transfer  statement:      goto  m 
where  m  is  an  integer. 

Conditional  transfer  statement:      if  p(x.  ,...,x  )  then  goto  n 

where  p  is  an  n-ary  predicate   (n>0; ,    x,  ,...,x^   are  variables,  and  m  is  an 
integer.      0-ary  predicates,   such  as  TRUE  and  FALSE  are  prohibited. 

Halt  statement:      stop. 

A  program   is  a   finite   sequence  of  legal   statements,   each   statement   prefixed   by 
a   unique   integer,    known   as  its  label.      We   shall   use   the   term   "transfer  statements" 
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whenever  we  wish  to  include  both  conditional  and  unconditional  transfers.   For 
every  transfer  statement  'goto  m'   or  'if  p  then  goto  id',  m  must  be  the  label  of 
some  statement  in  the  program.   That  statement  is  called  the  target  of  the  transfer 
statement.   Every  program  contains  exactly  one  start  statement  which  appears  as  the 
first  statement  of  the  sequence  and  may  not  be  the  target  of  a  transfer  statement. 
Every  program  contains  at  least  one  halt  statement.   The  final  statement  of  a 
program  must  be  either  a  halt  statement  or  an  unconditional  transfer. 

If  s  is  the  k^  statement  in  a  program  and  s  is  the  (k+1)^'-  statement,  then 

we  say  that  s  physically  precedes  s  ,  and  a  physically  succeeds  s  .  We  say  that 

statement  s  executlonally  precedes  statement  s   (s  executionally  succeeds  s  )  if 
1  ^—^ 2    2  ^ 1 

and  only  if  either  s  is  a  transfer  statement  (either  conditional  or  unconditional) 

and  s  is  its  target,  or,  s   is  not  an  unconditional  transfer  or  halt  statement, 

2  1 

and  s  is  its  physical  successor.   A  statement  s  is  syntactically  reachable  if  and 

only  if  there  is  a  sequence  of  statements  s  , .  . .  ,s   such  that  s   is  the  start 

1       n  1 

statement,  s   is  s,  and  for  each  i=1,...,n-l,  s  executionally  precedes  s 

n  i  i+1 

A  transfer  statement  is  called  ineffective  if  it  physically  precedes  its 
target.   All  other  transfer  statements  are  effective.   We  require  that  every 
statement  in  the  program  be  syntactically  reachable,  and  that  all  transfer 
statements  be  effective.   Violations  of  these  restrictions  are  at  best  the  product 
of  coding  practices  which  tend  to  obscure  program  logic  and  should  therefore  be 
eliminated.  More  significantly,  their  presence  may  well  be  indicative  of  certain 
types  of  logical  or  typographical  errors  (e.g.   incorrect  or  missing  labels; 
missing  statements).   It  seems  unlikely  that  a  programmer  would  intentionally  write 
code  which  can  never  be  executed  or  include  a  completely  unnecessary  transfer 
statement  to  the  very  same  statement  that  would  have  been  executed  without  the 
transfer.  Although  we  are  concemed  mainly  with  testing  as  a  means  of  uncovering 


program  errors,  it  Is,  of  course,  highly  "desirable  to  find  and  correct  as  many 
errors  as  possible  before  testing  begins:.   We  propose  that  the  procedure  described 
In  this  paper  include  as  part  of  its  output  some  indication  of  potentially 
troublesome  situations  encountered  in  processing  a  program,  similar  in  nature  to 
'syntax  error'  messages  produced  by  a  compiler.   We  will  therefore  continue  to 
mention  the  types  of  program  anomalies  which  may  be  discovered  at  each  stage  of  the 
procedure. 

Flow  Graph  Theoretic  Concepts 

A  program  can  be  uniquely  decomposed  into  a  set  of  disjoint  blocks  having  the 
property  that  whenever  the  first  statement  of  the  block  is  executed,  the  other 
statements  are  then  executed  in  the  given  order.   Furthermore,  the  first  statement 
of  the  block  should  be  the  only  statement  which  may  be  executed  directly  after  the 
execution  of  a  statement  in  another  block.   Formally,  a  block  is  a  maximal  set  of 
ordered  statements  br<s.  ,...,8  >  such  that  if  n>1,  for  1=2,..., n,  s.  is  the  unique 

«xecutional  successor  of  s .  ,   and  s.  ,   is  the  unique  executional  predecessor  of 

1-1       1-1 

8.  .  Thus  the  first  statement  of  a  block  is  the  only  one  which  may  have  an 
1 

executional  predecessor  outside  the  block,  and  the  last  statement  is  the  only  one 
which  may  have  an  executional  successor  outside  the  block.  Every  conditional 
transfer  must  be  the  last  statement  of  a  block,  since  effective  conditional 
transfers  cannot  have  unique  executional  successors. 

The  program  graph  G  representing  a  program  Q  consists  of  one  node  i 
corresponding  to  each  block  b.  of  Q  and  an  edge  from  node  J  to  node  k,  denoted 
(J,k),  if  and  only  if  either  the  last  statement  of  b.  Is  not  an  unconditional 
transfer  and  it  physically  precedes  the  first  statement  of  b  ,  or  the  last 
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statement  of  b  is  a  transfer  whose'  target  is  the  first  statement  of  b,  .   If  there 
is  an  edge  from  node  J  to  node  k  we  say  that  node  J  is  a  predecessor  of  node  k,  and 
k  is  a  successor  of  J.   The  node  ccrrtsponding  to  the  block  whose  first  statement 
is  the  start  statement  of  the  prograa  is  known  as  the  start  node.   Such  a  node  has 
no  predecessors.   A  node  corresponding  to  a  block  whose  final  statement  is  a  halt 
statement  is  known  as  an  exit  node  and  has  no  successors.   In  addition,  a  node  has 
two  successors  if  and  only  if  the  final  statement  of  its  corresponding  block  is  a 
conditional  transfer.   The  requirement  that  all  transfer  statements  be  effective 
guarantees  that  the  two  successors  are  different  nodes.  That  is,  for  every  pair  of 
nodes  1  and  J  there  is  at  most  one  edge  from  node  1  to  node  J. 

A  £ath  is  a  finite  sequence  of  nodes  (n  , ...,n  ),  k  >  2,  such  that  there  is  an 
edge  from  n  to  n   .  for  1=1 ,2,.  . .  ,k-1 .  Because  all  transfer  statements  must  be 
effective,  there  Is  at  most  cwie  edge  between  any  pair  of  nodes,  allowing  us  to 
represent  a  path  as  a  sequence  of  nodes,  rather  than  as  a  sequence  of  edges.  Note 
that  the  definition  of  path  is  a  purely  syntactic  one.  That  is,  a  path  is  any 
sequence  of  nodes  connected  by  edges.   A  complete  path  is  a  path  whose  initial  node 
is  the  start  node  and  whose  final  node  is  an  exit  node.   Note  that  it  may  be  the 
case  that  there  is  no  input  which  will  cause  the  sequence  of  statements  represented 
by  a  particular  path  to  be  executed.  Since  it  is  known  that  there  can  be  no 
algorithm  to  decide  whether  a  given  path  is  executable  [12],  we  do  not  require  that 
all  complete  paths  be  executable. 

A  syntactically  endless  loop  is  a  path  (n  ,...,n  ),  k>1,  n  =n,  ,  such  that  none 
of  the  blocks  represented  by  the  nodes  on  the  path  contain  a  conditional  transfer 
statement  whose  target  is  either  in  a  block  which  is  not  on  the  path  or  is  a  halt 
statement.   Such  a  loop  contains  no  possible  escape  and  can  be  detected 
algorlthmlcally  and  eliminated  from  a  prograun,  or  flagged  as  a  possible  error.  We 


therefore  assume  that  programs  contain  no -syntactical ly  endless  loops.   Since  all 
statements  in  a  program  must  be  syntac-tically  reachable  and  there  may  be  no 
syntactically  endless  loops,  we  are;  guaranteed  that  every  node  appears  on  some 
complete  path,  although  possibly  an  unexecutable  one. 

The  Def/Use  Graph 

Our  path  selection  criteria  are  based  on  an  investigation  of  the  ways  in  which 
values  are  associated  with  variables,  and  how  these  associations  can  affect  the 
execution  of  the  program.     This  analysis   focuses  on  the  occurrences  of  variables 
within  the  program;    the  actual    functions  and   predicates   to   be   computed   play   no 
role.     Each   variable  occurrence   is  classified   as  being  a  definitional  occurrence, 
computation-use  occurence,  or  predicate-use  occurrence.     We  shall  refer  to  these  as 
def ,  c-use  and  p-use   ,   respectively. 

The  assignment  statement    'y:rf(x    , . . . ,x  )'   contains  c-uses  of  x.  ,...,x 
followed   by  a  def  of  y.  " 

The  input  statement    'read  x    ....,x  '  contains  defs  of  x,  ,...,x  . 

1 '    '  n  In 

The  output  statement  'print  x.  ,...,x  '  contains  c-uses  of  x, ,...,x^. 

The  conditional  transfer  statement  'if  p(x  ,...,x  )  then  goto  m'   contains 

p-uses  of  X  , . . .  ,x  . 

1      n 

In  the  following  discussion  we  will  say  that  a  node  of  a  program  graph 
contains  a  c-use  or  a  def  of  a  variable  if  there   is  a   statement  in. the 
corresponding  block  containing  a  c-use  or  a  def  of  that  variable.     Because  the 
value  of  a  variable  occurring  in  the  predicate  portion  of  a   conditional   transfer 
statement  will   affect   the  execution   sequence  of  the  program,  we  associate  p-uses 
with  edges   rather  than  with  nodes.      If  the   final   statement   of  the  block 
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corresponding   to   node   i    is    'if  p(x, ,.,.,x    )  then  goto   m',   and    the   two    successors   of 
node   i  are    nodes   J  and   k   then  we  will   say  that  edges    (i,j)   and    (i,k)    contain  p-uses 
of  x-^, .  . .  ,x   .      In   figure    2,   node   6  contains  c-uses   of   z  and   x,    followed    by  a  def  of 
2,   followed   by  a  c-use  and   a  def  of  pow.      Edges    (5,6)   and    (5,7)   each   contain   a 
p-use  of  pow. 

Since  we   are   interested    in  tracing  the  flow  of  data  between  nodes,  any 
definition  which   is   used   only  within   the  node  in  which   that  definition   occurs   Is  of 
little   impxartance  to  us.     Thus  we  categorize  defs  and   uses  as  being  either  global 
or  local.     A  c-use  of  a  variable  x  is  a  global  c-use  if  and  only  if  there   is  no  def 
of  X  preceding  the  c-use  within  the  block  in  which  it  occurs.     That   is, the  value  of 
X  must  have  been   assigned   in  some  block  other  than  the  one  in  which  it  is  being 
used.     Otherwise  it  is  a   local  c-use.     Global  c-uses  are  often   cailled   locally 
exposed    uses   in   the  data   flow  analysis   literature    ([11]). 

Let  X  be  a  variable  occurring  in   a  program.  We   say  that  a  path 
(i,n,,...,n    ,J),  m   >^  0,  containing  no  defs  of  x  in  nodes  nj,...,n     is  called   a 
def-clear  path  with  respect  to   (wrt)   x   from  node   i   to  node   J.      A  path 
(i,n^,.,.,n   ,j,k),  m  2  0,  containing  no  defs  of  x  in  nodes  n    ,...,i^,J  is  called   a 
def-clear  path  wrt  x   from  node  i  to  edge  (J,k).     An  edge   (i,j)  is  a  def-clear  path 
wrt  X  from  node  1  to   edge   (i,J).     A  def  of  a  variable  x  in  node  1  is  a  global  def 
if  and  only  if  it   is   the  last  def  of  x  occurring  in  the  block  associated   with  node 
i     and    there  is  a  def-clear  path  wrt  x  from  i  to  either  a  node  containing  a  global 
c-use  of  X  or  to  an  edge  containing  a  p-use  of  x.     Thus,  a  global  def  defines  a 
variable  which  will  be  used  outside  the  node  in  which   the  definition  occurs.      A  def 
of  a  variable   x  in  node   i  which   is  not  a  global   def   is  a   local   def  if  and   only  if 
there   is  a  local  c-use  of  x  in  node  1  which  follows  this  def,  and  no  other  def  of  x 
appears   between   the   def  and    the  local   c-use.      The  def  of    'answer'   in  node   9  of 
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1.   start 

2.  read    x,y 

3.  if  y<0  then  goto    6 

ii.    pow   :=   y 

5.    goto    7 

6.  pow  :=   -y 

7.  z   :=    1 

8.  if  powrO  then  goto 

12 

9.    z  :=   z«x 

10.    pow   :=   pow-1 
11  .   goto    8 

12.  if  y>=0  then  goto 

13.  z   :=    1/z 

m 

m.   answer   :=   z+1 

15.  print   answer 

16.  stop 

read  x , y 


p  o  w :  =  3'^ 


pow : =-y 


y^  0 


answe  r : =  z  +  1 
print  answer 


figure  2 
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figure    2  is   local.      Any  def  which   is  neithe,r,  global   nor    local   will   never   be   used 
and    the  program   should  be   examined    for  possible   error. 

Methodologies   which   detect   program  anomalies   using  data   flow  analysis    [13. 1*^1 
consider  the  presence  of  any   def-clear   path  wrt  a   variable   x  from   the   start  node  to 
a  use  of  X  to  be  a  possible  error.    Since   some  of  these  paths  may   not   be   executable, 
there  may  well   be   no   error.      If,   however,   none  of  these  paths  contains  a   definition 
of  X,   and   at   least   one   is  executable,   then  there   is   indeed    an  error.      Thus   we 
assume  that  there  is  some  path  from  the  start  node  to  every  global  c-use  or  p-use 
of  a  variable  which  contains  a  def  of  that  variable.     Programs  which   violate  this 
assumption   should  be   flagged   as  having  a  possible  error. 

We   create  the  def/use  graph  from  a  program   graph  by  associating  each   node  i 
with   two   sets,  def  and   c-use,  and  each  edge   (i,J)  with   the  set  p-use.     def(i)   is 
the  set  of  variables   for  which  node  i  contains  a  global  def;   c-use(i)   is   the  set  of 
variables   for  which  node  i  contains  a  global   c-use;   p-u3e(i, J)   is   the  set  of 
variables   for   which  edge    (i,J)   contains  a  p-use.      An  edge    (i,J)   for  which 
p-use (i, J)   is  non-empty  is  called    a  labelled  edge;   if  p-use (i, J)   =  0  then    (i,J)   is 
called   an  unlabelled   edge.      Because   0-ary  predicates  are   not  allowed,  edges   which 
are   the   sole  out-edges  of  a  node  are   always   unlabelled,  while  those  which  are  one 
of  a  pair  of  out-edges  are  always    labelled.      In  figure   2  these   sets  are: 


node 

c-use 

def 

1 

e 

{x,y) 

2 

{y} 

{pow) 

3 

ly} 

{pow} 

4 

Jir 

{2) 

5 

9 

ss 

6 

{x,z,pow} 

{z,pow} 

7 

a 

f! 

8 

{z} 

[z] 

9 

{2) 

P" 

edge 


p-use 


(1,2) 

(y) 

(1,3) 

{y} 

(5,6) 

{pow} 

(5,7) 

{pow) 

(7,8) 

{y} 

(7,9) 

{y) 
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Note   that    'answer',   which   has  only   a  local    tier  and   a  local    c-use,   does   not  appear 
in   these   sets.    Edges    (2,iJ) ,  (S,'^) ,  ('<,5>r(6^."'50 ,   and    (8,9)   are   unlabelled. 

We  now  define   several  sets  needed   in'  the  construction  of  our  def/use   criteria. 

Let   i   be  any   node,   and   x   any  variable   such    that  xig  def(i).      Then  dcu(x,i)    is    the 

set   of  all   nodes   J    such    that   x£.c-u3e(J)  and   for  which    there    is  a   def-clear   path 

wrt  X  from  i  to   j;   dpu(x,l)   is   the  set  of  all  edges    (J,k)  such    that  x€  p-use(J,k) 

and  for  which   there   is  a   def-clear  path  wrt  x  from  1  to   j.     The  dcu  and  dpu  sets 

for   figure   2  are: 

dcu(x,1)={6)  dpu(x,1)=/ 

dcu{y,l)={2,3)  dpu(y,1)={(l,2), (1,3), (7,8), (7,9)1 

dcu(pow,2)={61  dpu(pow,2)={(5,6),(5,7)) 

dcu(pow,3)={6)  dpu(pow,3)=((5,6),(5,7)l 

dcu(z,U)  =  {6,8,9}  dpu(z,il)=0 

dcu(2,6)={6,8,9}  dpu(z,6)=^ 

dcu(pow,6)={6}  dpu (pow, 6) ={(5, 6), (5,7)} 

dcu(z,8)={9l  dpu(z,8)=^ 

Let  P  be  a   set  of  complete  paths   for  a  def/use  graph  of  a  given  program.     We 
say  that  a  node  i   is  included  in  P  if  P  contains  a  path   (n.  ,...,n^)  such   that   i=nj 
for  some   j,    1   <  J   <  m.     Similarly,  an  edge   (i^  ,1^  )   is  included   in  P  if  P  contains  a 

path   (n   ,...,n   )  such   that   i,=n.    and  i   =n         for  some   J,    1   <  J   <  n-1«     A  path 

1  m  Ij  2j+l 

(i,,...,!-   )  is  included   in  P  if  P  contains  a  path   (n^^ ,. ..  jn^j^)  and   i,  =nj  , 

1   =n  ,...,!  =n      ■  for  some   J,    1   <  J   <  n-k+l.     In  addition,  we   say  that  P  is 

2      j  +  1  k     j+k-1  -  ->  - 

executed  if  every  path  contained   in  P  is   traversed  during  the  course  of  executing 
the  program  on   a  set  of  test   input  data. 


A   Family   of  Path  Selection   Criteria      ..ir    r.:    .T 

We   now   introduce   a    family   of    path   selection   criteria.      Let   G    be   a   def/use 
graph,   and   P  be   a   set   of   complete  paths  of  G.      Then 

P  satisfies   the  all-nodes  criterion   if  every   node  of  G  is   included   in   P. 

P  satisfies   the  all-edges   criterion   if  every   edge  of  G  is   included   in  P. 

P  satisfies   the  all-defs  criterion   if  for   every   node  i  of  G  and  every 
x€  def (i),   P  includes  a  def-clear  path  wrt  x  from  i  to  some  element  of 
dcu(l,x)  or  dpu(i,x). 

P  satisfies   the  all-p-uses  criterion   if  for   every   node  i  and  every   x6def(i), 
P  includes  a   def-clear   path  wrt  x  from   i   to   all   elements  of  dpu(x,i). 

P  satisfies   the  all-c-uses/some-p-uses  criterion   if  for   every   node  i  and   every 
x£  def (i),    P  includes  some   def-clear  path  wrt  x  from   i  to   every   node   in 
dcu(x,i);      if  dcu(x,i)   is  empty,   then   P  must    include  a  def-clear   path   wrt  x 
from   i  to   some  edge   contained    in   dpu(x,i).      This  criterion  requires    that   every 
c-use  of  a  variable   x   defined    in  node  1  must    be   Included    in   some  path  of  P. 
If  there   is  no   such   c-use,   then   some  p-use  of  the  definition   of  x  in   i  must   be 
Included.      Thus    to   fulfill  this  criterion,  every   definition  which   is  ever  used 
must   have    some  use   included   in  the  paths  of  P,   with   the  c-uses   particularly 
emphasized  . 

P  satisfies   the  all-p-uses /some -c-uses  criterion   if  for  every   node  i  and  every 
xf  def(i),    P   includes  a   def-clear  path  wrt  x   from   1   to   all   elements  of 
dpu(x,l);      if  dpu(x,l)   is  empty,   then   P  must    include  a  def-clear  path   wrt  x 
from   1   to   some   node  contained    in   dcu(x,i).      As   in   the   case  of 
all-c-uses/some-p-uses,   this  criterion  requires  every   definition  which   is  ever 


-lo- 
used   to    be    used    in    some   path   of   P.    In    this   case    however,    the   emphasis    Is   on 
p-uses . 

P  satisfies  the  all-uses  criterion  if  for  every  node  i  and  every  xfdefd),  P 
Includes  a  def-clear  path  wrt  :i  from  1  to  all  elements  of  dcu(x,l)  ani  to  all 
elements  of   dpu(x,l). 

A  path    (nj,...,n|^)   is   loop-free   if  nl/  nj  whenever   1^  J.      P  satisfies   the 
all-du-paths   criterion   if  for   every   node   i  and   every   x£def(i),    P   Includes 
every  loop-free   def-clear  path   wrt  x  from  i  to   all   elements  of  dpu(x,i)  and    to 
all   elements  of  dcu(x,l). 

P  satisfies    the  all-paths  criterion   If  P   Includes  every   complete  path  of  G. 
Note   that,   due   to  loops,  many  graphs  have   infinitely  many  complete  paths. 


Criterion   c.    includes  criterion   c„  if  for   every   def/use  graph  G,   any  set   of 

complete  paths  of  G   that  satisfies   c     also  satisfies   c    .      Criterion  c, 

1  2  ' 

strictly  includes  criterion  C2,  denoted  c.  =  >c„,  if  and  only  if  c  includes  c^  ,  and 
for  8one  def/use  graph  G  there  is  a  set  of  complete  paths  of  G  that  satisfies  c- 
but  not  Cj.  Note  that  this  is  clearly  a  transitive  relation.  We  say  that  criteria 

Ci  and  Co  are  incomparable  if  neither  c  =>c  nor  c  =>c  . 

I  ^  1: J      2  21 

We   can  assume   that  all   def/use  graphs  contain  more   than  one   node.      Single-node 
graphs  have  only  one  path  and    thus  any  of  the  criteria  would   select   that   path. 
Furthermore  we  may   assume   that  all   def/use  graphs  have  more   than  two   nodes  and   at 
least   two   labelled   edges.      This   follows  inmediately  from  our  definition  of   block, 
and    the  requirement   that  all  transfer  statements  be  effective. 
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Theorem: 

The   family   of  criteria   is   partially  ordered   by  strict   inclusion   as   shown    in 
figure    3.      Furthermore,   criterion   c.    strictly  includes  criterion   c.    if  and   only  if 
it   is  explicitly  shovm   to   be   so   in   figure    3  or   follows   from  the   transitivity   of  the 
relationship. 

Proof;    In  most  cases   the   inclusion   is   imnediate  and   will   not   be   discussed.      In  such 
cases  we  restrict  our  proof  to  the  demonstration  that  each  inclusion  shown  in 
figure   3  is  In  fact  a   strict  inclusion,  and   that  pairs  of  criteria  not  shown  in 
figure   3  to  be  related  by  strict  inclusion  are   incomparable. 

1 .  all-paths  ->   all-du-paths 

Let  G   be  any  graph   containing  an  infinite  number  of  paths.     G   can  contain   only 
a  finite  number  of  different   loop-free  paths.     Let  P  be  the  smallest  set  of 
complete  paths  containing  all  loop-free  paths  of  G.     By  definition,   P 
satisfies   the  all-du-paths  criterion.     Since  P  can  contain  at  most   one 
complete  path  for  each  loop-free  path   (othervise  there  would  be  a  smaller  set 
of  complete  paths  containing  all  loop- free  paths),   P  must   be  finite,  and   hence 
does  not  satisfy  the  all-paths  criterion  for  G. 

2.  all-du-paths  =>  all-uses 

Consider  the  graph  of  figure   «*.      {( 1 ,2,H,5,7) ,  (1 ,3f '^.S,?) )  satisfies  all-uses, 
but  not  all-du-paths,  since  it  does  not  Include  the  path   {^  ,2,^,6,7) ,  which 
includes  a  def -clear  path  wrt  y  from  node  2  to  node  7. 

3.  all-uses   =>   all-p-uses/some-c-uses 

Consider  the  graph  of  figure   5.      f  (1 ,2,3,l<) ,  (1 ,3,5)  1  satisfies 
all-p-uses/some-c-uses,    but  not  all-uses,   since  there  is  no  def-clear  path  wrt 
y  from   the   def  of   y  in  node  2  to   the   c-use  of  y  in  node   5. 
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all-c-ases/ -■    ■■-'- 
some-p-uses 

/  ^     ^ 

all-  ==^  all-du  =^all-  all- 

patha     paths     uses  defs 


■**  all-p-uses/ 


-p. 
some-c-uses 


all-p-usea  =^  all-edges  =^  all-nodes 


figure  3 


lb  - 


y  :=0 


code ; =y+z 
print  code 


figure  4 


19 


read  x,y 


y:=y+x/2 


y:=-y 
print  y 


print  y 


figure  5 
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k.      all-uses   r>  all-c-uses/soine-p-uses 

Consider  the  graph   of   figure    6.      {(1,3))   satisfies 

all-e-uses/some-p-uses,    but  does  not  satisfy  adl-uses   since   there    is  no   path 

containing  the  p-use  of  x  in  edge   (1,2). 

5.  all-c-uses/some-p-uses   =>  all-defs 
all-p-uses/some-c-uses   =>  all-defs 

Consider  the  graph   of   figure   7.      {(1,2))  satisfies  all-defs,    but  does  not 
satisfy   all-c-uses/some-p-uses  or   all-p-uses/some-c-uses. 

6.  all-p-uses/some-c-uses  =>  all-p-uses 

Consider  the  graph   of   figure   8.      {( 1 ,2,  3,5) ,  (1  .S,**) )   satisfies 
all-p-uses,    but   not  all-p-uses/some-c-uses,   since   dpu(y,2)   is  empty,    but   there 
is  no  def-clear  path  wrt  y  from  the  definition  of  y  in  node  2  to  the  c-use  of 
y  In  node  *». 

7.  all-p-uses   =>  all-edges 

In  this  case,  the  Inclusion   does  not   follow  immediately  from  the  definitions. 

Let  P   be  any  set   of  complete  paths  of  a  def/use  graph  G   that  satisfies 

all-p-uses.     We  will  show  that  every  edge   (i,j)  in  G  is  included   in  P. 

Case    1:    (1,J)   is   labelled. 

Then   (i,j)   contains  a  p-use  of  some  variable,  say  x.      Since  this 
p-use  must    be  preceded   by  a  def  of  x  along  some  path  from  the   start  node 
to    (1,J).  P  must   include  a  def-clear  path  wrt  x  from  that  def  to    (i,J). 
Since    (i,j)  is  an  edge  along  that  path,   P  includes    (i,J). 

Case   2:    (i,J)   is  not   labelled. 

Case   2a:    i   is   the   start  node. 

Then   edge    (i,J)  must    be   the  first   edge   in  every  complete  path  of  G. 
Since  G   has  at   least   one   labelled   edge,    P  contains  at    least   one 


2  1  - 


X  <  0 


print  'error' 


read  x 
print  X 


x>  0 


print  y 


figure  6 
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Y;=1  -X 
print  y 


x<  0 


3)  y:  =  J3r 
print  y 


figure  7 
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X  integer 


y;=yX 


read  x  y 


X  not  integex 


print  y 


print  * error* 


figure  8 
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complete  path,   and    therefore   P   includes    (i,J). 
Case   2b:    1   has  at   least    one  labelled   In-edge. 

This  means   that   i   has  a   predecessor   k,   and   edge    (k,i)   is    labelled. 
Since   J  is   the  unique   successor  of   i,   any  complete  path   containing 
(k,i)  must  also  contain    (i,J).      As    (k,i)   is  a   labelled   edge,   it  must 
be   included   in  P,   and,   therefore,   so   must    (i,J). 
Case  2c:    1   has  only  unlabelled   in-edges. 

The  definition  of    'block'  insures  that   if  any  node  has  only  one 
in-edge,   that  edge   is   labelled,  and   we  may  therefore  assume   that   i 
has  more  than  one  unlabelled   in-edge.     Thus   there     must   be  at   least 
two  distinct  paths   from  the  start  node  to   i.      At   least   one  of  these 
paths  must  contain  a  labelled  edge   (otherwise  the  paths  vrould  be 
identical).     Select  any  such  path  pr(s,. . .  ,n^,n    ,, . .  ,1),  where  s  is 
the  start  node   (s  may  be  n.),  edge   (n    ,n    )  is   labelled,  and 
(n    , ...,i)   contains  only  unlabelled   edges.      Any  complete  path 
containing  (n    ,n   )  must  contain    (n    ,..<.,i),  and   since  i  is   the 
unique  predecessor  of  j,  edge    (l,j)  must   be  on  that  complete  path  as 
well.     Because    (n   ,tl  )   is   labelled,  it  must   be  included   in  P,  and    . 
therefore   so  is    (i,J). 
We  now  demonstrate  that  the  inclusion   is  strict.     Consider  the  graph  of  figure 
9.      {(1,2,3,2,'*)  1  satisfies  all-edges  but  not  all-p-uses  as  it  does  not 
include  a  def-elear  path  wrt  x  from  the  def  of  x  in  node  1  to  edge    (2,'J). 

8.     all-edges  =>  all-nodes 

Consider  the  graph  of  figure    10.      {(1,2,3))  satisfies  all-nodes  but  not 
all -edges. 
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figure   10 


-  27  - 

9.  all-c-uses/3ome-p-uses  and  all-p-uses/soroe-c-uses  are  incomparable 

The  graph   of   figure    5  demonstrates   that  all-p-uses/soroe-c-uses  does  not 
include  all-c-uses/some-p-uses  .      { (  1 ,2,  S,'^) .  ( 1 .  3,5)  }   satisfies 
all-p-uses/some-c-uses,    but  not  all-c-u'ses/some-p-uses  since  there   is  no 
def-clear  path  wrt  y  from  the  def  of  y  in  node  2  to  the  c-use  of  y  in  node  5. 
Similarly,    the  graph   of   figure   6  demonstrates   that 

all-c-uses/some-p-uses  does  not  include  all-p-uses/some-c-uses.      {(1,3)} 
satisfies  all-c-uses/some-p-uses,    but  not  all-p-uses/some-c-uses  since  there 
is  no  path  containing  the  p-use  of  x  in  edge   (1,2). 

10.  all-defs  and   all-p-uses  are   incomparable 
all-defs  and  all-edges  are   incomparable 
all-defs  and  all-nodes  are   incomparable 
all-c-uses/some-p-uses  and  all-p-uses  are   Incomparable 
all-c-uses/some-p-uses  and   all-edges  are   incomparable 
all-c-uses/some-p-uses  and  all-nodes  are   incomparable 

The  graph  of  figure   6  demonstrates   that  all-c-uses/some-p-uses  does  not 
include  all-nodes.      {(1,3)}   satisfies  all-c-uses/some-p-uses,    but  does  not 
include  node  2.      Because  of  the  transitivity  of  inclusion,  this  also  means 
that  all-c-uses/some-p-uses  does  not  include  all-edges  or  all-p-uses,  and 
that  all-defs  does  not  include  all-p-uses,  all-edges,  or  all-nodes.     The 
graph  of  figure   9  demonstrates   that  all-p-uses  does  not  include  all-defs, 
since    {(1,2, 3, 2,^4),  (1,2,^4)}  satisfies  all-p-uses,    but  the  def  of  y  in  node  3 
is  not  used   along  either  path.      Because  of  the  transitivity  of  inclusion, 
this  also  means   that  neither  all-edges  nor  all-nodes  includes  all-defs,  and 
that  all-p-uses,   all-edges,   and   all-nodes  do  not   include 
all-c-uses/some-p-uses. 
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Analysis  of  the  Criteria 


The   criteria  all-nodes    (statement  coverage)   and   all-edges    (branch   coverage) 
are  often   used    in   program   testing  despite  the  fact   that   they   are   extremely  weak 
criteria.      Our  search   for   stronger  criteria   that  make  use  of  data  flow  information 
led    us  at    first   to   all-defs.      Our  assumption   is   that   every   definition   in   a  program 
was   included   by  the  programmer   because   it  vrould  eventually   be   used    somewhere     and 
thus  a   well-tested   program   should   test  all  definitions.   However,   we  rejected 
all-defs  as  an  adequate  criterion  since  it  does  not  even   include  all-nodes.    In   [15] 
errors  are   separated  into  domain  errors,  which  occur  when  an  incorrect  path  is 
chosen   due   to   a  control    flow  error,   and    computation  errors,  which   occur  when   a 
correct   path   is  chosen   but  an  assignment  statement  contains  an  erroneous 
computation,      All-defs  can  detect  computation  errors   but  not  necessarily  domain 
errors,   while  all-edges  can  detect  domain  errors   but  not  necessarily  computation 
errors.      In  looking  for  criteria   that  can  detect   both    types  of  errors,  we   separated 
uses  of  variables   into   p-uses  and   c-uses.      All-p-uses   is  our   first  data  flow 
analysis  criterion  which   includes  all-edges,    but   it     too      primarily   detects  domain 
errors.      It   is   stronger  than  all-edges   since   it   requires  a   path   from  every 
definition  of  a  variable  to   every  possible  p-use  of  that  variable,  while  all-edges 
merely  requires   that  there  be  some  path   that   includes   that  p-use.     Since  the  value 
of  a  variable  contained   in  a  predicate  may  have  been  defined   in  one  of  several 
places,   it   is  clear  that  all-p-uses  can  uncover  more   errors   than  all-edges.      The 
criterion   all-p-uses/some-c-uses   is   the  weakest   of  our  criteria   that   includes   both 
all-defs  and  all-edges.     We  are  guaranteed   that  testing  according  to  this  criterion 
exercises  every   edge     and   every  computation. 

Consider  the  program  of  figure    11,  which   is  a    translation   into   our   programming 
language  of  the  Wensleysqroot  program   presented    in    [7]   to  compute  V^,   0  £  p  <    1 ,   to 
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accuracy   e,   0  <   e   <    1.      The  program   contains  an  error;    statements    11   and    12  should 
be   interchanged.      The   set   of  paths    { ( 1 , 6) , ( 1 ,2, 3,^,2, 3, 5,2,7) )   satisfies  all-edges, 
but  would  not  detect   the  error.      As   stated    in    [7],    "a  looping  factor   of  two    is 
required    to   derive   test  data   that   reveals    the  bug,"   that   is,   the  tester  must 
specify  that  some  path   containing  at    least   two    executions  of  the  loop   be   tested. 
In  fact,   two   or   more   executions   of  the  loop  may   not  suffice.    The  problem   is   that 
the  definition   of  c  in  node  5  is  never  used   unless  the  set  of  paths  includes  a 
definition-clear  path  wrt  c  from  node  5  to   node  3,  and    thus   the  error  cannot  be 
detected    unless   the  path    (5,2,3)   is  included.     The  all-defs  criterion,  however, 
does  require   that  all  definitions  be  used,  and   thus  any  set  of  paths  selected 
according  to  this  criterion  would  have  to   include  (5,2,3).     The  set  of  paths 
{(1,2,3,5,2,3,5,2,7) ,  (1,2, 3, '1,2, 3,^,2, 7) }  would   detect  the  error;    however  node  6 
and  edge    (1,6)   would  not   be   tested.      All-p-uses   is  not  adequate  to   find    the  error 
either.    {( 1 ,6) ,  (1 ,2,7) ,  (1 ,2,3,5,2,7) ,  (1 ,2,3, H,2,3,'»,2,7) }  satisfies 
all-p-uses  without   including     (5,2,3).     However,  since   the  program  contains  no 
p-uses  of  the  definitions  of  c  in  nodes   i\  and   5,  all-p-uses/some-c-uses  does 
require   that  the  paths    (5,2,3)   and    (^,2,3)    be  included. 

Since  the  value  of  a  variable  used   in  a  c-use  may  have  been  defined   in  one  of 
several   places,  all-c-uses/some-p-uses   Is  more  likely  to   find   computation  errors 
than  aill-p-uses/some-c-uses.     In  particular,  all-c-uses/some-p-uses  requires  paths 
between  every  definition   and  every  possible  c-use  of  that  definition.     For   figure 
11,  this  means   that  any  set  of  paths  chosen  according  to  all-c-uses/some-p-uses 
must   include  the  paths    (i<,2,3,*t),    (*<,2,3,5),    (5,2,3,^)  and    (5,2,3,5).     However, 
this  criterion   does  not  include  all-edges  either.      For  example, 

{(1,2,3,5,2,3,5,2,7),(1,2,3,^,2,3,5,2,7),(1,2,3,'<,2,3,'<,2,7),(1,2,3,5,2,3,'<,2,7)} 
satisfies  all-c-uses/some-p-uses,    but  does  not   include  edge    (1,6).    Furthermore,   it 
does  not   include  path    (1,2,7),   which   would   be   executed   if  the   input  data  were 


-    30   - 


■  2  then  goto    l8 
e  then  goto    16 


1.  start 

2.  read  p,e 

3.  d:=1 
U.  x:=0 

5.  c:=2«p 

6.  if  e  > 

7.  if  d  < 

8.  d:=d/2 

9.  t:=c-(2«x+d) 

10.  if  t   <   0  then   goto    1^1 

11.  x:=x+d 

12.  c:=2e(c-(2«x+d)) 

13.  goto   7 
1^4.  c:=2«c 

15.  goto   7 

16.  print   x 

17.  stop 

18.  print    'error' 

19.  stop 


print 'error-' 


print    X 


figure    11 
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incorrect  and   e>l.     Since   the  program  does  check  for  p< 1 ,    it  may  very  well   be   an 
error   that   it  does  not  explicitly  check  for   e  £.  1 .      This  error  vrould  be   detected   by 
all-p-uses/some-e-uses,  however,   since   it  doe*   require    that  the  path    (1,2,7)    be 
included. 

The   criterion   all-uses,  which   includes   both  all-p-uses/some-c-uses  and 
all-c-uses/some-p-uses,  can  detect  both   types  of  errors.     This  criterion   is  similar 
to  required  element  testing   [16].     One  set  of  paths  which   satisfies  the  all-uses 
criterion   for   figure    11  is   {( 1 ,6) , (1 ,2, 3,5,2, 3,5,2,7) , (1 ,2,7) , (1 ,2, 3,1,2, 3,5,2,7) , 
(1, 2, 3, *», 2, 3, 1,2, 7),  (1,2, 3, 5, 2, 3,^4, 2, 7)).     Notice  that  any  set  of  paths  which 
satisfies   this  criterion  must  contain  the  paths    (1,5),    (1,2,7),  and   all  of  the 
combinations  of  predicates   represented   by  the  paths    (1,2,3,1),    (1,2,3,5),    (5,2,3,1) 
and    (5,2,3,5).      However,   for   some  programs,   this  criterion  may   not  test  all 
possible  combinations  of  predicates,  as  we  saw  in  figure   1.  We  therefore   defined 
the  all-du-paths  criterion. 

Conclusions  and  Future  Work 

The  data  flow  criteria  that  we  have  defined    can  be  used   to  bridge  the  gap 
between   the  requirement  that  every   branch  be   traversed   and    the  requirement  that 
every  path  be   traversed.     Our  criteria  focus  on  the  interaction  of  portions  of  the 
program  linked  by  the  flow  of  data  rather  than  solely  by  the  flow  of  control. 
Research  is  currently  underway  to  determine   the  relative  costs  of  the  criteria  in 
terms  of  the  number  of  test  cases  required    to  satisfy  them,  and   to  more  precisely 
characterize   the   types  of  errors  detectable  by  each.     We  envision   a  tester  being 
able   to  select  a  particular  criterion  by  determining  whether  the  likely  payoff  in 
terms   of  errors  detectable   is  worth   the  added    cost    in   terms   of  additional   tests 
necessary. 
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